How ESE Works

Before changes are made to a database file, ESE writes the changes to a transaction log file. After a change has been safely logged, it can then be written to the database file. It is common for these changes to become available to users just after the changes have been secured to the transaction log, but before they have been written to the database file.

ESE employs a sophisticated internal memory management system that is tuned for high performance and can efficiently manage the caching of dozens of gigabytes (GBs) of database pages. Therefore, physically writing out changes to the database file is a low-priority task during normal operation.

If a database suddenly stops, cached changes are not lost just because the memory cache was destroyed. When the database restarts, ESE scans the log files, and reconstructs and applies any changes not yet written to the database file. This process is called replaying log files. The database is structured so that ESE can determine whether any operation in any log file has already been applied to the database, needs to be applied to the database, or does not belong to the database.

The following sequence describes this process in more detail:

When an operation occurs, the page that requires updating is placed in the ESE cache and the log buffer records the operation in memory.
The changes are recorded by the database engine but not immediately written to disk. These are known as "dirty" pages because they have not been committed to the database.
As the database pages are changed, the log buffer is notified to commit the change, and the transaction is recorded in a transaction log file (which may or may not require a log roll; that is, the creation of a new transaction log file).
The dirty database changes are flushed to the database file.
The log file checkpoint is advanced.

Buffer Cache Management

ESE uses a comprehensive buffer manager to decide how much memory the buffer cache should use via an internal feature called Dynamic Buffer Allocation (DBA). ESE continuously monitors the state of the cache, verifying system requirements, and adjusting cache size when necessary. DBA uses the following rules to determine how the buffer cache should grow or shrink over time:

The more memory available, the faster the cache grows.
The more cache memory, the faster the cache shrinks.
The higher the memory load, the faster the cache grows.
The higher the available memory load, the faster the cache shrinks.

The CygNet DBS-based services use the following service configuration keywords to define the memory limits that the system can allocate for the disk cache. The default value for these keywords is different for each DBS-based service and is set in the service’s configuration file.

Keyword	Description
DBS_DISK_CACHE_IN_MEGS	DBS_DISK_CACHE_IN_MEGS specifies the maximum amount of memory that the system will allocate for the disk cache in megabytes. See Buffer Cache Management for more information. The DBS_FILE_CACHE_MAX (DBS file cache max) info item will report the value of this keyword. This database caching behavior takes advantage of the Windows File System Cache if the specified cache size is less than 10% of the sum of the database and index files (as calculated at startup). If this is the case, a line will be logged at startup in the service’s log file which contains "Enabling usage of the filesystem cache." The File System Cache is managed by the operating system and is used by many different processes. It allows each process to compete for the limited memory available on the machine based on need and priority. The benefit is that if you have a server with a lot of extra memory, it can be temporarily put to use as a database cache until a process with higher priority needs it. The only potential downside is that if the File System Cache grows to use all available memory, and you want to run a new process, it may take a little longer than usual for that process to start, as the memory needs to be reclaimed from the File System Cache. If this delay becomes an issue, you can avoid using the File System Cache by setting the DBS_DISK_CACHE_IN_MEGS value to more than 10% of the sum of the database and index files, or contact CygNet Support for assistance in limiting the maximum size of the File System Cache.
DBS_MIN_DISK_CACHE_IN_MEGS	DBS_MIN_DISK_CACHE_IN_MEGS specifies the minimum amount of memory that the system will allocate for the disk cache in megabytes. Controls how eager the DBS’s data cache will be to consume memory in the absence of other memory pressure in the system.
DBS_FILESYSTEM_CACHE_USAGE	DBS_FILESYSTEM_CACHE_USAGE allows a DBS to configure its database engine with a small dedicated cache and allows Windows to donate spare memory to further improve the caching of database data. See Buffer Cache Management for more information about Windows File System Cache usage. Set to ON to enable the usage of the Windows File System Cache. Set to OFF to disable the feature entirely. Set to HEURISTIC to follow the 10% model used by the DBS_DISK_CACHE_IN_MEGS keyword, described here.

Keyword

Description

DBS_DISK_CACHE_IN_MEGS

DBS_DISK_CACHE_IN_MEGS specifies the maximum amount of memory that the system will allocate for the disk cache in megabytes. See Buffer Cache Management for more information.

The DBS_FILE_CACHE_MAX (DBS file cache max) info item will report the value of this keyword.

This database caching behavior takes advantage of the Windows File System Cache if the specified cache size is less than 10% of the sum of the database and index files (as calculated at startup). If this is the case, a line will be logged at startup in the service’s log file which contains "Enabling usage of the filesystem cache." The File System Cache is managed by the operating system and is used by many different processes. It allows each process to compete for the limited memory available on the machine based on need and priority. The benefit is that if you have a server with a lot of extra memory, it can be temporarily put to use as a database cache until a process with higher priority needs it.

The only potential downside is that if the File System Cache grows to use all available memory, and you want to run a new process, it may take a little longer than usual for that process to start, as the memory needs to be reclaimed from the File System Cache. If this delay becomes an issue, you can avoid using the File System Cache by setting the DBS_DISK_CACHE_IN_MEGS value to more than 10% of the sum of the database and index files, or contact CygNet Support for assistance in limiting the maximum size of the File System Cache.

DBS_MIN_DISK_CACHE_IN_MEGS

DBS_MIN_DISK_CACHE_IN_MEGS specifies the minimum amount of memory that the system will allocate for the disk cache in megabytes. Controls how eager the DBS’s data cache will be to consume memory in the absence of other memory pressure in the system.

DBS_FILESYSTEM_CACHE_USAGE

DBS_FILESYSTEM_CACHE_USAGE allows a DBS to configure its database engine with a small dedicated cache and allows Windows to donate spare memory to further improve the caching of database data. See Buffer Cache Management for more information about Windows File System Cache usage.

Set to ON to enable the usage of the Windows File System Cache.
Set to OFF to disable the feature entirely.
Set to HEURISTIC to follow the 10% model used by the DBS_DISK_CACHE_IN_MEGS keyword, described here.

ESE evaluates the availability of system memory resources before saving pages in the page cache. If memory is highly constrained, the cache will not grow even if the maximum setting has not been reached.

The VHS has its own service configuration keywords for configuring these parameters. See VHS Database Page Cache for more information.

Keyword	Description
VHS_MAX_CACHE_SIZE_MB	Specifies the maximum size of the database page cache for the VHS datastore in megabytes. The default for this keyword is 1000 megabytes.
VHS_MIN_CACHE_SIZE_MB	Specifies the minimum size of the database page cache for the VHS datastore in megabytes. The default for this parameter is 200 megabytes.

Transactions

ESE is a sophisticated, transaction-based database engine. A transaction is bundle of one or more database update operations that are treated as an atomic (indivisible) unit. The term "update" is used generically to describe any operation that changes the database: updating, adding, or deleting records. All operations in a transaction are both completed and permanently saved, or no operations are performed.

In CygNet, a transaction could be a single operation (for example, updating a PNT record) or a series of operations (for example, creating or deleting a device). Be aware, though, that a seemingly single operation may actually be a series of operations, particularly if it involves updating indexes. For example, adding a PNT record would include the operations for adding the record and updating the index.

ESE transactions are highly reliable and are referred to as ACID transactions because they are identified by the following attributes: atomic, consistent, isolated and durable. See ACID for more information. Such transactions allow CygNet to retrieve data only from reliable data states and maintain data consistency in the event of an unexpected process termination or system crash.